However, these models suffer from quadratic computational cost in the input sequence lengthn to compute pairwise attention in each layer.

In this work, we investigate an alternative setting for tuning regularization parameters, namely data-driven algorithm design, following the previous line of work by Balcan et al. [

Deep learning practice is increasingly driven by powerful foundation models (FM), pre-trained at scale and then fine-tuned for specific tasks of interest.

The optimization phase in deep learning consists in minimizing an objective function w.r.t. the set of

Inthis work, we propose analgorithm-software co-designed sparse convolution based onanovelout-vector-wise (OVW) sparse pattern.

We introduce a semidefinite programming hierarchy to estimate the global and local Lipschitz constant of a multiple layer deep neural network.

Thus our heuristic relaxation does give valid upper bounds of the exact Lipschitz constant.